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The Council of Chief State School Officers (CCSSO) conducted an alignment content 
analysis of the 2009 National Assessment of Educational Progress Reading and Science 
Assessments administered in 2009 for students at grades 4 and 8. The analysis was 
conducted using the methodology of the Surveys of Enacted Curriculum (SEC). The 
study of NAEP Reading and Science Assessments was supported by the National Center 
for Education Statistics through the Assessment Division. This report on the findings of 
the analysis was prepared by staff of the Council of Chief State School Officers, led by 
the project director Rolf K. Blank. 

Executive Summary 

Beginning in summer 2009, the complete set of NAEP student assessment items for 
grades 4 and 8 Science and Reading 2009 assessments were analyzed for comparison to 
the NAEP Item Specifications which are based on the NAEP Assessment Erameworks for 
these subjects (National Assessment Governing Board, 2009). The data produced from 
the Assessment analysis can be used for multiple comparisons to other assessments or to 
state or national standards for learning. The review and analysis of NAEP Assessment 
items was led by CCSSO and the arrangements for item access were made through the 
National Center for Education Statistics, Assessment Division. The analysis was 
conducted with the SEC methodology developed by Porter and Smithson (2001) which 
has been used in analyses of many state, national and international assessments through 
the SEC State Collaborative managed by CCSSO (see www.SECsurvev.org) . 

The most general summary measure of alignment between two documents analyzed with 
the SEC method (either assessment or standards) reported is the alignment index (AI), a 
scale value that ranges from 0.00 to 1.00, with 1.00 indicating perfect alignment between 
the two documents being compared. In this report on the alignment of NAEP 
assessments, three types of criteria are utilized to adjudicate the sufficiency of the 
alignment results: (1) the national average of alignment indices for state assessments to 
state standards, based on results available from the SEC database www.SEConline.org , 

(2) the alignment index for assessment instruments that were previously analyzed through 
NCES, and (3) an expected state alignment threshold level that is consistent across all 
subjects and grade levels, based on our prior data and experience with alignment analyses 
across states and including national and international documents. The standard baseline 
criterion for the overall alignment index (AI) is 0.25, and the criterion is 0.50 for each of 
three sub-measures of alignment (Balance of Representation, Categorical Concurrence, 
and Cognitive Complexity). 

Results: Alignment of NAEP Assessments to Item Specifications 

Science. Two NAEP assessment grades, 4 & 8, were analyzed for science. Alignment 
results for both science assessment grades indicate good alignment between the NAEP 
assessment instruments and the Item Specifications {Grade 4 alignment index = 0.37, 
Grade 8 alignment index = 0.42). The NAEP science alignment index compares 
favorably across alignment criteria, i.e., whether compared to the standard baseline 
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alignment (AI = 0.25), or previous NAEP assessments alignment analysis (e.g., NAEP 
Gr. 4 Math AI = 0.40, Gr.8 Math AI = 0.36). 


Analysis Methodology and Procedures 

The SEC alignment analyses for NAEP science and reading assessments were conducted 
with procedures developed by Andrew C. Porter and John E. Smithson for describing and 
examining relationships between documents related to subject-matter content and/or 
reports of instructional practice (Porter & Smithson, 2001; Smithson & Porter, 2004). 

The basic steps in the SEC method are as follows. Eirst, content review and coding to the 
SEC frameworks are carried out for each of the documents being analyzed (generally, 
standards (or specifications) and assessments). The content review and coding is 
generally done by four content analysts who all have strong experience in their subject 
and teaching the subject, and have been trained in the SEC content analysis procedures 
(Smithson, 2007). In general, the process asks analysts to describe and code references to 
subject matter content in terms of two dimensions— topics (what students should know), 
and expectations for student performance (what students should be able to do). All 
measurements are made at the intersection of these two dimensions. That is, every 
reference to topic is paired with a category of student performance expectation. This 
paired description is referred to as a content code. The observed topics and expectations 
are coded into the SEC framework for the subject. Eor assessments, the unit of analysis is 
an assessment item. Content analysts are permitted to use up to three content codes to 
describe the instructional content assessed by a test item. Eor content standards the unit 
of analysis is either a standard strand or sub-strand. Content analysts are permitted to use 
up to six content codes to describe a content standard strand. 

Analysts all have subject matter content expertise, and are typically drawn from 
assessment or curriculum specialists at the state or regional level as well as university 
faculty. Analysts undergo a four-hour training process, which includes sample coding 
items and group discussion about coding rationales and convention. Coding teams 
consist of 3-5 members. All analysts make independent decisions and submit 
independent data collection reports. A consensus model is not employed. The strategy 
instead is to collect a comprehensive description of content from multiple perspectives. 
Analysts discuss assessment items and standards text as determined necessary by the 
team in order to discuss the rationale and options for content descriptions used to describe 
a specific unit. While members discuss their coding rationales and decisions, and 
analysts are free to make changes in their coding selections as the result of team 
discussion, analysts are instructed to make independent decisions based on their own 
individual professional judgment, bindings from generalizability studies previously 
conducted indicate that to obtain reliable analysis results a minimum of three content 
analysts are needed for each team, and four members are preferred (see, Smithson, 2007, 
for a description of content analysis procedures). 

Content codes are processed and averaged across analyst teams to construct a content 
description of each document. These content analysis results can be arrayed into a two 
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dimensional matrix, with columns to represent each of the five categories of student 
performance, and rows to match the number of topics for the given content language. 

The content language matrix used for the analysis and coding is from the Surveys of 
Enacted Curriculum instruments (Blank, Porter, & Smithson, 2001). Currently a K-12 
taxonomy is employed in four subject areas; mathematics, science, English language arts 
& reading, and social studies. This report presents results for mathematics, science and 
language arts & reading only. 

Alignment analysis results are reported using four distinct summary measures: 
Categorical concurrence, Balance of representation, Cognitive complexity. Alignment 
index. Each document pair is summarized using each of these four measures at an overall 
level (i.e., across the entirety of both documents), and at more detailed, ‘sub-levels’ 

(based on curriculum content areas). The definition and purpose of each of these four 
measures are described below. 

Alignment Measures and Criteria 

Categorical Concurrence 

The most straightforward criterion in measuring alignment is the criterion which Webb 
(1997) calls "categorical concurrence". Here, the operational question is, "Does this 
assessment item fit one of the categories identifiable in the standards being employed?" 

If yes, we can say the item is "aligned". If we can say yes to every such assessment item, 
we could say that the assessment is aligned to the standards on the topic dimension. 

While the descriptive data collected during the content analysis process is reported at the 
intersection of topic and cognitive demand, analyses can be conducted on each dimension 
separately. These marginal measures yield descriptions of relative emphases for topic 
coverage and categories of cognitive demand that can then be compared in order to 
examine alignment strength in terms of one dimension or the other. Categorical 
concurrence is the summary measure that focuses on topic coverage. (The table below 
provides an example of a statistical table format for reporting the alignment measures.) 
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Categorical concurrence reports on the extent to which topics emphasized in the relevant 
state content standards are similarly emphasized in the state assessment. Results are 
reported for each content area, as well across the entire set of standards and assessment 
instrument. Categorical concurrence measures serve to identify areas where there is a 
mismatch between topics tested and those emphasized in the standards. 

Balance of Representation 

Categorical concurrence as a sole criterion has significant shortcomings in using it as a 
measure of alignment. An assessment that focused exclusively on say, one standard to 
the exclusion of all the rest, would be equally well aligned as an assessment that provided 
equal representation for each standard. A measure based on categorical congruence alone 
could not distinguish between the two, though the two tests would be dramatically 
different in the range of content assessed. 

This leads to balance of representation as a second criterion forjudging alignment: a 
sense of range, or breadth of coverage. An assessment can only test a portion of the 
subject matter that is presented to students. It is important then that assessments used for 
accountability purposes present some balance of representation across the range of topics 
that students are expected to be proficient in. An alignment measure that speaks to this 
balance of representation allows investigation into the relationship between the subject 
matter range identified in the content standards and the range of topics represented by a 
particular test. 

Balance of Representation measures the relative emphasis of standards and assessments 
by content area. [ Science The mathematics content language consists of 7 content areas 
(see Table 4) while English Language Arts & Reading is made up of 14 content areas 
(see Table 3). ] The balance of representation measure reported for each content area is 
simply the difference between the relative proportions of emphasis focused on a given 
content area when comparing the standards to the test. A positive number represents 
over-emphasis on the test, while a negative number indicates under-emphasis of tested 
content. The overall balance of representation measure is based on the aggregate of 
these content area measures and is converted to report the total proportion of content held 
in common across the two documents. 

Cognitive Complexity 

While balance of representation is an improvement over simple categorical congruence, it 
is becoming increasingly clear that the depth of coverage, or cognitive complexity of the 
content provided to students in one or another subject area represents an important 
ingredient to their success on a given assessment (Gamoran, et.al, 1997). Cognitive 
complexity refers to the performance goals or cognitive expectations of instruction, and 
provides a third criteria or dimension to include in calculating an alignment measure. 

Cognitive complexity examines the level of alignment between standards and 
assessments on the dimension of cognitive demand, or expectations for student 
performance. As with the other alignment measures, results are reported at the level of 
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content area, as well as an overall measure. A low measure on eognitive eomplexity 
indieates a mismateh between the state content standards and the state assessment on the 
distribution of emphasis aeross the five categories of eognitive demand. 

Alignment Index 

The alignment index incorporates elements of all the previous measures into a single 
index for deseribing the alignment between two eontent deseriptions. This measure is 
based on eell-by-eell eomparisons aeross all eells used for the measure. In the ease of the 
‘overall’ alignment index, this value is based on comparisons across 520 cells for K-8 
Mathematies, and 575 cells for English Language Arts & Reading. As a result, the 
alignment index is the most sensitive of the alignment measures, taking into 
eonsideration both topie eoverage and the eonfiguration of eognitive demand, or student 
performanee in determining the level of agreement between two content descriptions. It 
is typically the smallest value when eompared to the other alignment measures. The 
alignment index is reported at an overall level, as well as by individual eontent area. 

Eaeh SEC reporting display comparing two documents, either as a eontour map or tile 
ehart format, has an alignment index (AI) statistie reported, and the initial ehart ineluding 
all main topies has a “eoarse grain” index reporting the eonsisteney between the main 
topics. 

All alignment summary measures have a range of 0.00 to 1.00. Eor eaeh measure, a 
result of 1.00 would indieate perfeet alignment between the two doeuments on the 
construct of the measure being reported. Each measure reports a proportion of content 
held in eommon between two eontent deseriptions (i.e., eontent standards or 
assessments). The phrase ‘held in eommon’ here has speeifie meaning, referring to a 
proeess of eomparing two values and seleeting the smaller of the two for plaeement into a 
counter that aggregates aeross comparisons. Each alignment measure involves some 
number of such comparisons, with the resulting summary measure calculated by sum of 
all interseets (smaller of the two values) aeross the eomparisons employed for a partieular 
measure. 

Determining Acceptable Levels of Alignment 

The proeedures utilized for this analysis provide a systematie, quantitative, reliable and 
replicable methodology for describing and analyzing relationships between eontent 
deseriptions. What the methodology has not previously offered are clear criteria for 
determining what eonstitutes “good” alignment. Today, with results available aeross 
several years from more than two-dozen states, three eriteria ean be reasonably 
established. 

An important and unique feature of the eontent analysis proeedures used for these 
analyses is that the resulting eontent deseriptions allow eomparisons of alignment aeross 
states (see www.SEConline.org) . With content analyses having been conducted on 
standards and assessment documents in more than two dozen states (many through the 
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SEC State Collaborative), it is possible to examine alignment measures in order to 
establish an average measure of alignment between eontent standards and assessments for 
a given subjeet, aeross several states. Summary alignment measures can thus be 
described as being above, at, or below the average alignment for a particular subject 
across all states for which relevant data are available. 

Another strategy for establishing alignment criteria would be to simply set a threshold 
value that can be considered to represent a minimally acceptable level of alignment 
consistent for all grades and subjects based in part on alignment results seen across states, 
grade levels and subject areas. For example, for this and previous alignment reports, 
balance-of-representation measures reported for each content area are highlighted if the 
measure exceeds an absolute value of 0.05. A content area with a difference in the level 
of emphasis between the standard and the assessment that exceeds 0.05 suggests an 
imbalance in the breadth of content assessed by the test. The actual threshold value 
selected (0.05) is itself somewhat arbitrary, and is therefore primarily intended as a 
diagnostic indicator useful for identifying areas that state assessment and standards 
developers may wish to examine in order to inform any future adjustments in either the 
assessment instrument or standards document. 

Additionally, now that a significant number of documents have been analyzed across 
many states and subjects for several years, it seems reasonable to set specific values for 
each of the four summary alignment measures that indicate a minimal threshold for 
‘good’ alignment. For the purposes of this report, these threshold criteria have been set at 
0.25 for the alignment index, and 0.50 for each of the other three summary alignment 
measures (balance of representation, categorical concurrence and cognitive complexity). 
These values are very close to current national average alignment index across states in 
mathematics (avg.: 0.26, range 0.1 to 0.55) and science (avg.: 0.22, range 0.1 to 0.5), but 
are noticeably higher than the national average for English Fanguage Arts & Reading 
(avg. 0.17; range 0.08 to 0.35). 

A third criterion against which to compare the current set of alignment results comes 
from the alignment results for previous state assessments, NAEP assessments or other 
assessments including TIMSS and PISA. Several of these comparisons with other studies 
are reported below. 

SEC Content Analysis process with NAEP 

The SEC instruments include a two-dimensional content framework for each subject that 
was designed to collect, analyze and report data on curriculum that has been taught and 
analyze curriculum content in relation to standards (intended curriculum) as well as 
assessments that determine what has been learned. During the summer 2009, CCSSO 
arranged for science and reading content coding teams to use the SEC method and 
content framework to analyze the NAEP science and reading assessment items for 2009. 
A four-person specialist coding team was assigned each grade level for science and 
reading. Additionally, reading and science specialist teams analyzed and coded the 
NAEP item specifications documents which are based on the NAEP assessment 
frameworks (National Assessment Governing Board, Assessment and Item Specifications 


8 



for the 2009 Reading National Assessment of Educational Progress; Assessment and 
Item Specifications for the 2009 Science National Assessment of Educational Progress 
http ://www .nagb .org/publications/frame works .htm) . 

CCSSO together with our research contractor Wisconsin Center for Education Research 
(WCER) carried out alignment analyses using the coding results and this report presents 
the results of the analysis regarding the alignment of NAEP Reading and Science to the 
Item Specifications, and the alignment of NAEP assessment to state standards for these 
subjects. 

The Surveys were developed by CCSSO and the Wisconsin Center for Education 
Research through a collaborative project involving educators, researchers, and subject 
area specialists (see Blank, Porter, & Smithson, 2001; Smithson & Porter, 2004). The 
SEC analysis method has been used previously to analyze standards and assessments in 
mathematics, science, English language arts reading, and social studies in over 30 states 
(see www.SEConline.org for further description of the methodology, details about the 
SEC content frameworks, and state content analysis results). CCSSO conducted an 
alignment analysis of NAEP Mathematics assessments (Blank & Smithson, 2008). The 
SEConline.org website also provides data for previous analyses of international 
assessments (such as TIMSS and PISA), and analysis of standards and assessment from 
other countries. These data and analyses are open to the public. 

The alignment analysis data and alignment statistics (including alignment index) are 
presented through graphic displays using Content Maps (see results below) and 
Marginals summary charts. Also available for review are the content analysis data for 
each of the analysts and the averages used in producing the alignment results. Statistics 
of alignment are also computed for each of the three criteria of alignment, categorical 
concurrence, balance of representation, and cognitive complexity, and these tables are 
available in appendices from CCSSO. 

Inter-rater Reliability 

The SEC inter-rater reliability for the 4-person teams which produced the SEC content 
analyses are well above average. Inter-rater reliability provides a good indication of how 
well analysts agreed in their content review and coding results. The average inter-rater 
reliability for science grades 4 and 8 is 0. 7. 

Method for NAEP Item Specifications Analysis 

The content analysis work conducted on the NAEP Item Specifications required some 
departure from the normal content analysis procedures as a result of the item 
specifications not specifying content at the intersection of science topic and cognitive 
demand. In the NAEP Item Specifications documents (National Assessment Governing 
Board, 2009) webpage, we found that these two dimensions are specified in separation 
from one another. To accommodate translation into the SEC content language a 
computational algorithm was employed to distribute emphases across categories of 
cognitive demand based on the text of Chapter 3 in the Item Specifications, and Exhibit 
1 3 in the Science Eramework. Since no declarative statement could be found in either 
document regarding the desired distribution of emphasis across cognitive demand 
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categories, the specific values employed is somewhat arbitrary. For the purposes of 
reporting the content analysis results here, the values used for distributing emphasis 
across the five categories of cognitive demand were as follows: 10% Recall, 20% 
Procedural, 30% Communicating Understanding, 30% Analyzing Information, and 10% 
Evaluate/Critique . 

Review of NAEP analyses 

Following the initial reporting of content analysis charts and data by WCER, CCSSO 
contracted with an independent specialist in each subject, science and reading, to conduct 
a review of the analysis results and the consistency of the data. The reviewers have 
excellent knowledge and experience with the NAEP frameworks and the assessments, 
and also had experience with the SEC methodology and frameworks. The reviews were 
submitted to NCES to provide an independent view of the process and data, and they 
provided strong endorsement of the SEC methods and results. 


Findings: Results of Alignment Analysis 

Science. Two NAEP assessment grades, 4 & 8, were analyzed for science. Alignment 
results for both science assessment grades indicate good alignment between the NAEP 
assessment instruments and the Item Specifications (Grade 4 alignment index = 0.37, 
Grade 8 alignment index = 0.42). The NAEP science alignment index compares 
favorably across alignment criteria, i.e., whether compared to the standard baseline 
alignment (AI = 0.25), or previous NAEP assessments alignment analysis (e.g., NAEP 
Gr. 4 Math AI = 0.40, Gr.8 Math AI = 0.36). 

Readers Guide : Two types of alignment analysis results are reported in the charts: 
“content maps” and “content marginals”. Both provide descriptive information about 
content analysis results based upon data collected through the analysis of NAEP 
assessments and specifications. Content maps display results of content analyses with 
surface area maps using a systematic taxonomy for describing subject-matter content. 
The darker colors represent greater concentration at the intersection of topic and 
expectations (or cognitive demand). Content marginals report the results for degree of 
emphasis on each topic and expectations (cognitive demand) across all content areas. 
Content marginal results are displayed using simple bar charts. A content map of one 
main topic is reported at the fine grain level. 

The “coarse grain” alignment statistic is the degree of alignment between the two maps 
shown using only the main topics. The “alignment index” is the degree of overall 
alignment at the fine grain and coarse grain levels. 

Science Grade 4 Alignment analysis results 

• Alignment index = 0.37, Coarse grain alignment (main topics) - 0.54 

• Topics in NAEP assessment grade 4 (right content map) emphasized are: Nature 
of science. Measurement, Botany, Properties of Matter 
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• The NAEP assessments include items assessing all 5 levels of expectations, with 
greater emphasis on Procedures/conduct investigations in science and 
Communicating Understanding of science knowledge. 

• The science topic Measurement is highlighted, and the SEC analysis shows the 
sub-topic Data Displays are emphasized in grade 4 NAEP assessment. 
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Science Grade 4 Alignment analysis 
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Marginals analysis Grade 4 Science 


Topic Coverage and Cognitive Engagement 
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Science Grade 8 Alignment analysis results 

• Alignment index = 0.42, Coarse grain alignment (main topics) = 0.5 1 

• Topics in NAEP assessment grade 8 (right content map) emphasized are: 
Ecology, Earth Systems, Meteorology 

• The NAEP assessments include items assessing all 5 levels of expectations, with 
greater emphasis on Memorize/Recall and Communicating Understanding of 
science knowledge. 

• The science topic Earth Systems is highlighted, and the SEC analysis shows the 
sub-topics Mineral/rock formation and Plate tectonics are emphasized in grade 8 
NAEP assessment. 
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Alignment Analysis with States 

For the purpose of comparing NAEP content analysis to states standards, CCSSO is 
reporting content maps and alignment to NAEP with the states that have used the SEC 
methodology. We are reporting the alignment analyses according to three categories: 
a) states with high NAEP achievement scores than the national average, b) states not 
significantly different from the average, and c) states below the national average. 

The content maps demonstrate the degree of alignment of NAEP assessments with state 
standards. These examples highlight the kind of analyses that are possible for state level 
standards and assessments with the NAEP assessments. 

On the attached charts, the “alignment overall index” refers to the degree of consistency 
or match between the content (2 dimensions) for the document on the left side with the 
content of the document on the right side. The “coarse grain” statistic refers to the 
alignment or consistency of the main topics and expectations for the two documents 
shown in a chart. 


Patterns of alignment of State Standards to NAEP Assessments 

• The Alignment Index for relationship of State standards to NAEP varies from 0.2 
to 0.3; The Coarse Grain alignment from varies 0.35 to 0.45 

• NAEP assessments were found to include more science content topics and the 
assessments covered more of the five types of expectations for students than the 
State standards at both grades 4 and 8 

• State standards generally place more emphasis on Nature of Science and Science 
and Technology at grades 4 and 8 than the NAEP assessment 

• The NAEP assessment places more emphasis on Measurement at grade 4 than 
states, and more emphasis on Earth Systems and Meteorology at grade 8 than 
states. 
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State Science Standards 
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Science Grade 4 NAEP by State Standards 
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Summary 

Using the SEC analysis methods and content frameworks, CCSSO has been able to report 
on the degree of alignment of NAEP assessments to the NAEP item specifications. We 
are confident the results of the CCSSO analysis provide a valid description of the content 
assessed by the NAEP assessments and a valid method for comparing NAEP to state 
standards for student learning. The results of the NAEP content analysis will be posted 
on the SEC website, and they will be available for use by educators, leaders, and 
researchers to conduct further studies or comparisons of NAEP assessments to other 
standards and assessments. The data and results of this study can address several kinds of 
questions: 

• What is the extent of content knowledge and skills a student needs to do well on 
NAEP assessment as compared to the knowledge and skills needed for an 
assessment based on state standards? 

• What are the different content topics covered by NAEP assessments at grades 4 
and 8, as compared to standards of states? 

• What are the expectations, or cognitive demands, required for students to perform 
well on NAEP as compared to assessments based on state standards? 

• If states or others want to revise their curriculum and instruction towards 
improved performance on NAEP, what are the specific content areas and methods 
of teaching and learning that will need to receive focus? 
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Access Online 


The results of the NAEP Science and Reading Assessments content analysis are available 

on the SEC online system at 

http ://seconline . wceru w .or g/sec W ebHome .htm 

Directions: at this site, click on Content Analysis; then, 

“Eor access to content maps of Standards and Assessments analyzed thus far,” click here ; 
then Select Science (or ELA), K-12, and Submit. 

Select: NAEP 2009 Grade 4 or Grade 8 using pull-down menu in left column and the data 
can be compared to any state standards or national or international standards or 
assessment in the right column. 


External reviewers: 

Science: Brett Moulding, former science supervisor and State curriculum director, Utah 
State Department of Education, Experience: NAEP Science Assessment review 
committee (ETS), NAEP Science Erameworks development steering committee and state 
review (2005), SEC content analysis and coding for states and international assessments. 

Reading: Julie Harper, Delaware state department of education EEA and reading 
specialist, NAEP Reading assessment review committee, SEC experience in content 
analysis EEAR K-12 
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